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Abstract 

Background: Although considerable evidence exists about the effectiveness of audit coupled with feedback, very 
few audit-with-feedback interventions have been done in either home care or supportive living settings to date. 
With little history of audit and feedback in home care or supportive living there is potential for greater effects, at 
least initially. This study extends the work of an earlier study designed to assess the effects of an audit-with- 
feedback intervention. It will be delivered quarterly over a one-year period in seven home care offices and 11 
supportive living sites. The research questions are the same as in the first study but in a different environment. 
They are as follows: 

1. What effects do feedback reports have on processes and outcomes over time? 

2. How do different provider groups in home care and supportive living sites respond to feedback reports based 
on quality indicator data? 

Methods: The research team conducting this study includes researchers and decision makers in continuing care in 
the province of Alberta, Canada. The intervention consists of quarterly feedback reports in 19 home care offices 
and supportive living sites across Alberta. Data for the feedback reports are based on the Resident Assessment 
Instrument Home Care tool, a standardized instrument mandated for use in home care and supportive living 
environments throughout Alberta. The feedback reports consist of one page, printed front and back, presenting 
both graphic and textual information. Reports are delivered to all employees working in each site. The primary 
evaluation uses a controlled interrupted time-series design, both adjusted and unadjusted for covariates. The 
concurrent process evaluation includes observation, focus groups, and self-reports to assess uptake of the feedback 
reports. The project described in this protocol follows a similar intervention conducted in our previous study, Data 
for Improvement and Clinical Excellence-Long-Term Care. We will offer dissemination strategies and spread of the 
feedback report approach in several ways suited to various audiences and stakeholders throughout Alberta. 

Significance: This study will generate knowledge about the effects of an audit with feedback intervention in home 
care and supportive living settings. Our dissemination activities will focus on supporting sites to continue to use 
the Resident Assessment Instrument data in their quality improvement activities. 



Background 

Although there is evidence about interventions to 
improve quality of care in some healthcare settings, the 
evidence for quality-improvement interventions in long- 
term care (LTC) is variable [1-7]. Specific to audit and 
feedback interventions, there is little evidence on utility 
of this type of intervention in continuing care settings. 
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With unprecedented growth in both home care (HC) 
and supportive living (SL), data about the quality of care 
as well as about how to improve quality of care in these 
settings are a priority. Subsequently, we did a study on 
an audit-with-feedback intervention across the three 
streams of continuing care: LTC, HC, and SL. The pro- 
tocol we report in this paper is on the second phase of 
our study that will be carried out in HC and SL. The 
study protocol for the first phase in LTC was previously 
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reported [1]. In this second phase, we expand the origi- 
nal protocol to cover HC and SL settings. 

The number of healthcare resources needed to meet 
the needs of the large and ever-growing HC and SL sec- 
tor populations is a top priority of healthcare organiza- 
tions today. HC is defined as an array of services 
designed to meet individual and family needs [8,9]. SL is 
continuing care, usually in congregate living settings, 
and is also referred to as assisted living or designated 
assisted living in some sectors. The services vary from 
setting to setting to a degree but are primarily home 
care and support services to seniors. The Canadian 
Home Care Association reported that from 1995 to 
2006, the number of publicly funded HC clients grew by 
nearly 100% to approximately one million clients at any 
given time [9-12]. Further, the Health Council of 
Canada reported that from 1995 to 2002, the number of 
Canadians receiving publicly funded HC increased by 
60%. The need to deliver quality care to this growing 
population is essential and must be done in an efficient 
and effective manner in order to maximize our limited 
healthcare resources [9,11]. In the Government of 
Alberta's strategy, Aging in the Right Place, various 
initiatives are outlined to address the increasing health- 
care needs of HC and SL clients [13]. The strategies 
contained within Aging in the Right Place involve 
enhancing HC and SL through improved assessment, 
expansion of current programs, introduction of appro- 
priate emergency supports, and facilitation of transitions 
from acute care back into the community [13]. One out- 
come is that people receive the right care at the right 
time by the right provider. These actions are further 
supported in the Alberta Continuing Care Association 
document on implementing recommendations of the 
Ministry Advisory Committee on Health (MACH) 
[14,15]. Within the MACH report, there is an emphasis 
on incorporating the pillars of acceptability, accessibility, 
appropriateness, effectiveness, efficiency, and safety 
through the use of evidence-based research into SL 
facilities [14]. In order to mobilize an effective health- 
care workforce, it is essential that evidence-based prac- 
tice be at the forefront to ensure a high quality of care 
for HC and SL clients. In order to adequately attain a 
relevant and meaningful level of knowledge among com- 
munity healthcare providers, HC and SL client assess- 
ment data must be incorporated into research-based 
knowledge-translation (KT) strategies [16]. We position 
quality-improvement initiatives, such as audit and feed- 
back, as a KT strategy. 

Despite the aforementioned literature, there is a 
dearth of evidence from HC and SL settings as these 
sectors have received relatively little attention in terms 
of implementation of evidence-based practice, or at least 
little that is reported in the literature. Further, very little 



research has been done to examine the utility of client 
assessment data on improving nursing care in the HC 
and SL sectors. The Resident Assessment Instrument- 
Home Care (RAI-HC) tool belongs to a suite of tools 
used by organizations to collect, analyze, and understand 
the health status of clients. The RAI tools help to stan- 
dardize client assessment and form an evidence base to 
influence clinical practice and policy decisions and thus 
have been mandated by many organizations [17]. Within 
community settings, such as HC and SL, the RAI-HC is 
a useful tool for highlighting themes related to function- 
ing and quality of life of clients [18]. The RAI-HC qual- 
ity indicators can be assessed and recorded and include 
categories such as pain, falls, or depression [18]. The 
assessment data contained within the RAI-HC is easily 
accessible and, therefore, can be used to assess trends or 
patterns in client quality indicators. 

The detection of patterns and trends within a particu- 
lar RAI-HC data set can be further utilized to describe 
areas in which quality-of-care improvements could be 
made through increased use of evidence-based practice. 
In order to increase use of evidence-based practice and 
subsequently improve care in HC and SL, the relation- 
ships between the assessment findings and their useful- 
ness and applicability, including how well they are 
understood by healthcare providers, must be explored; 
one way to do this is through the use of audit and feed- 
back as a quality-improvement strategy. 

Primary purpose and study objectives 

The primary purpose of the Data for Improvement and 
Clinical Excellence (DICE) project is to assess the effects 
of an audit with feedback intervention in all three 
streams of continuing care: LTC, HC, and SL. The LTC 
intervention has been recently completed and is in the 
monitoring phase [1]. The HC and SL audit-with-feed- 
back intervention will be delivered quarterly over a one- 
year period in seven HC offices and 11 SL sites across 
Alberta, using data from the RAI-HC. 
We address these research questions: 

1, What effects do RAI feedback reports have on pro- 
cesses and outcomes over time? 

2. How do different provider groups in HC and SL 
respond to feedback reports based on RAI-HC quality 
indicator data? 

Methods 

The overall intervention evaluation uses a controlled 
interrupted time-series design with quarterly feedback 
reports in the seven HC offices and 11 SL sites across 
Alberta. Follow-up surveys to assess uptake of the audit- 
with-feedback intervention will be offered to all partici- 
pating employees one week after feedback report distri- 
bution. The purpose of the follow-up survey is to assess 
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staff response to the feedback reports and their intent to 
change their behavior based on what they learned from 
the feedback. We will not be measuring actual change 
in behavior through the surveys but rather their intent 
to change their behavior. This is similar to the survey 
tool that was used in the Data for Improvement and 
Clinical Excellence-Long-Term Care (DICE-LTC) study, 
with only wording revised to suit the HC/SL environ- 
ment. A sample survey instrument can be found in 
Additional File 1. 

We will use in-person audit and feedback in two 
zones and electronic distribution in two zones. This will 
be more fully described later in this protocol paper. 

The process evaluation will be conducted concurrently 
with the prospectively collected survey data and will 
consist of observation, focus groups, and self-report 
measures to assess uptake of the feedback reports. We 
define uptake as reading the feedback reports, discussing 
with colleagues and managers, and reporting some 
degree of intention to change behavior based on the 
reports. Observation will occur in the sites where we 
deliver the feedback reports in person. The focus groups 
will be targeted sessions based on the findings of the 
observations and the self-report survey and will be held 
in four of the participating sites. 

We received ethics approval from the University of 
Alberta Health Research Ethics Board, panel B. We 
obtained operational approval for all sites from all parti- 
cipating study organizations. Participation by staff is 
voluntary and will not affect their employment in any 
way. 

Project team 

The project team comprised both researchers and deci- 
sion makers who had existing relationships prior to this 
project, primarily through the Knowledge Brokering 
Group (KBG). The KBG team was focused on using 
data to inform decision making in continuing care set- 
tings. The overall makeup of the team and each mem- 
ber's background is described in the DICE-LTC 
protocol paper [1]. The specific program funding for 
this project requires active collaboration between 
researchers and decision makers [19], and the team 
works on a linkage and exchange, integrated KT model. 
New team members were added for this second phase 
in HC and SL, and again, the researcher lead and deci- 
sion-maker partners had existing collaborative relation- 
ships. We provide an updated description of the project 
team for DICE-HC in Additional File 2. 

Settings and sample 

The settings are seven HC offices and 11 SL sites across 
Alberta, Canada. The selected HC sites are part of 
Alberta Health Services (AHS) and their direct care 



providers and managers are employees of AHS, whereas 
the SL sites are operated by provider organizations 
under contract to AHS and the direct care providers 
and managers are employees of the provider organiza- 
tion. However, in all sites, case management and assess- 
ments are carried out by case managers who are part of 
AHS. The sites have all implemented the RAI-HC 
instrument [20]. 

Recruitment 

We recruited HC offices and SL sites in rural and urban 
areas across four zones in Alberta (North, Central, Cal- 
gary, and South) to participate [21]. Ideally, by the time 
we begin feedback reports, clients/residents of these HC 
office and SL sites would have been assessed using RAI- 
HC at least once and preferably more than once. The 
degree of RAI-HC implementation guided our choice of 
sites, as most HC and SL sites in Alberta began imple- 
mentation of RAI-HC in 2007. The roll-out was stag- 
gered across the province and was fully implemented by 
July 2011 and, going forward, all new client assessments 
and re-assessments will be completed using the RAI- 
HC. We recruited HC offices and SL sites with at least 
10 nonregulated staff, as well as at least 20 to 30 cli- 
ents/residents being assessed using the RAI-HC tool in 
order to obtain a critical mass. We recruited direct care 
employees from all of the professions, disciplines, and 
groups in the participating sites. Participants may 
include site managers, case managers, registered nurses, 
licensed practical nurses, healthcare aides, occupational 
therapists and assistants, recreational therapists and 
assistants, physical therapists and assistants, pharmacists, 
and social workers. 

The intervention 

Procedures for feedback report generation and distribution 

Feedback report distribution and data collection will 
occur in two ways: either in person or electronically. 
This was done for two reasons: first, logistically given 
the geographic size of Alberta and the desire to include 
most zones and, second, to assess the difference in 
response and utility between in-person and electronic 
feedback reports. We will use the Alberta Context Tool 
(ACT) to assess context in the sites as we did in DICE- 
LTC [1,21,22]. The ACT will be administered at the 
beginning of the project immediately following in-per- 
son information sessions and prior to beginning report 
distribution. Information sessions and administration of 
the ACT tool during the information sessions will occur 
in-person for all sites across the province. 

Unlike previous studies using feedback for quality 
improvement, this study focuses on the individual rather 
than the whole organization [23], and front-line staff are 
targeted to receive the feedback reports directly. Thus, 
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our purpose is to provide the feedback reports, gener- 
ated using RAI-HC data, directly to front-line staff using 
either an in-person distribution approach (North and 
Central zones) or via electronic distribution (Calgary 
and South zones). The RAI-HC covers a wide range of 
process and outcome data at the individual client level, 
and assessments are updated annually for each client 
unless there is a change in a client's health status or 
their functional or cognitive status, or if the client has 
been admitted to hospital. The quality indicators we 
have chosen to include in the feedback report are nega- 
tive mood, risk of falls, pain, delirium, and visits to the 
hospital in the last 90 days. These five areas are the top 
domains identified as important by HC and SL staff. 
They were selected in 2010 using a voting process com- 
pleted by AHS senior leadership, who consulted with 
their managers and/or direct care employees. The RAI- 
HC data will also be used to measure client-level 
outcomes. 

Data will be extracted by AHS for each of the partici- 
pating sites. All personal identifiers will be removed 
prior to extraction at AHS and delivery to the research- 
ers. The data extraction for each quarter will include all 
assessments completed within the prior 12-month per- 
iod so that in each quarterly report we will capture one 
year's worth of assessment data. The anonymized data 
will be submitted to the research team who will gener- 
ate feedback reports. 

The feedback reports are full color line graphs with 
minimal text on one double-sided page. An example of 
a feedback report is provided in Additional File 3. The 
reports will be distributed quarterly for one year. The 
first and third quarterly reports will include pain, nega- 
tive mood, and falls. The second and last quarterly 
reports will also include delirium and visits to the hospi- 
tal in the last 90 days. Each report will have the site's 
own data, with aggregate data from all other sites to 
compare their performance on the specific quality indi- 
cators to all other participating sites. 

We will distribute feedback reports to all available 
employees at the participating sites once each quarter. 
We elected to do quarterly feedback reports rather than 
monthly feedback reports for two reasons. First, it is 
unlikely that there will be enough new or repeat assess- 
ments completed each month to yield enough data for 
reliable estimates. Second, based on our observations in 
DICE-LTC, we did not want to burden participants with 
monthly reports. 

The following is a description of the in-person distri- 
bution approach, followed by the modifications that we 
will make for the sites that are participating electroni- 
cally. All of our visits will be prearranged with the site 
champions and during a time where we are most likely 
to capture the most staff on-site, for example, 



immediately following a team meeting when all or most 
staff come together at the SL site or HC office. In SL 
sites, there is often a huge campus, or several small 
sites, so it is not practical to try to reach staff outside of 
prearranged times. It is a similar case in HC, where staff 
are not in a central location, except during team meet- 
ings or in-services. 

Once every three months (i.e., every quarter), two 
research assistants (RAs) will hand deliver the report to 
all employees present and available at each site during 
the time he or she is there. The reports are specific to 
the HC or SL site, and all direct care providers and 
managers within these sites are targeted to receive the 
report that gives them information about how the cli- 
ents in their site compare to the other sites in the study 
on the selected indicator areas. Additional copies will be 
left in a central location, and staff will be told that they 
are welcome to take one later if they would like. The 
RAs will stay for about one hour after distributing 
reports to answer any questions or take comments. One 
of the RAs will also passively observe what employees 
do with the reports after receiving them and will discre- 
tely note how many times the employees read the 
reports, discuss them with others, discuss with the RA, 
or other actions such as throwing the reports away. No 
names or identifying information will be noted. We will 
use a simple check sheet as well as write narrative notes 
about the specific observations. This is the basis for our 
field notes and will be used in conjunction with staff 
self-report on the post-feedback survey to obtain an 
estimate of report uptake. At the end of the site visit, 
we will also post flyers reminding staff that we will be 
conducting surveys the following week. 
Process evaluation The purpose of the process evalua- 
tion is to assess both feedback report uptake and inten- 
tion to change behavior. This component of our 
evaluation will assess whether staff read and understood 
the report and whether they intend to change a particu- 
lar behavior that was targeted on the feedback reports. 
As in DICE-LTC, about a week after we distribute the 
feedback reports, we will ask employees to complete a 
survey to assess their response to the feedback reports. 
Staff will self-report on whether they received the 
report, whether they read the report, how they might 
use the feedback reports, whether they intend to use the 
information from the report to change care practices, 
and what sorts of changes they might consider making 
to their care practices as a result of the reports. All of 
these questions relate to feedback report uptake. The 
final section of the survey was constructed to capture 
intention to change behavior in pain assessment prac- 
tice, according to the constructs described in a manual 
on survey development using the Theory of Planned 
Behavior [24,25]. This section will be completed by the 
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direct care provider staff only. Surveys will be conducted 
in a room, away from client/resident care and activity. 
All surveys will be anonymous. We will ask some demo- 
graphic information, including the office/site they work 
in and their years of experience in HC/SL settings. This 
is similar to the tool that was used in DICE -LTC, with 
only wording revised to suit the HC/SL environment. 
After completing the survey, each employee will also be 
offered a $5 Tim Horton's gift certificate in appreciation 
for their participation. 

We will deliver the surveys to the sites during a time 
they are already meeting for another purpose. We will 
provide a self-addressed return envelope for returning 
the surveys to the research team. Our site champions 
are instrumental in the implementation of this study. 
Site champions will distribute the feedback report and 
survey to all employees who are not in attendance at 
the time of in-person report distribution using their 
usual mail distribution system. Those absent employees 
will review the feedback reports and complete the sur- 
vey and return it to the research team in the self- 
addressed envelope provided. 

In addition, we will conduct several 45-minute focus 
groups with regulated and unregulated healthcare provi- 
ders at selected sites to further explore and verify the 
quantitative findings from the post-feedback surveys. 
The intention of the focus groups is to delve deeper 
into how the respondents felt they understood the 
reports and their perceptions of report utility. We will 
conduct the focus groups with 5 to 10 healthcare provi- 
ders per group, which will take place after the report/ 
survey distribution in the third quarter. The focus 
groups will use a semi-structured discussion format. We 
will derive the questions to stimulate discussion based 
on our preliminary survey data from the first two quar- 
ters, discussing in more depth several variables that 
were addressed in the quantitative analysis related to the 
perceptions of the feedback reports. 
Procedures for electronic feedback report distribution 
We will follow the same order as with North and Cen- 
tral zones described above, but rather than in person, 
we will distribute feedback reports and conduct the 
post-feedback report survey electronically. We will use 
the continuing care desktop (CCD), a tool developed by 
the Centre for Health Evidence in collaboration with 
Alberta Health and Wellness that is available to all 
employees in continuing care throughout the province 
of Alberta [26]. Access to the CCD is through individual 
log-in privileges managed through the workplace. We 
plan to post the feedback reports by site on the CCD, 
with a separate notice emailed to the site leader. The 
notice will include a poster that the site leader will be 
asked to print and post in at least three prominent loca- 
tions to notify the employees to review the feedback 



report on the desktop. One week later, we will follow 
the same procedure with the site leader with a poster 
notifying the employees that the feedback report follow- 
up survey is available for them to complete. The partici- 
pating employees will log onto the CCD, where they will 
see a link to complete the online survey. By using an 
electronic distribution mechanism, we will be able to 
compare, descriptively, how electronic feedback reports 
compare to in-person feedback reports. As well, com- 
parisons on distribution for post-feedback report surveys 
will also be made between in-person and electronic 
formats. 

Analysis 

Our primary analysis, using time series with and with- 
out adjustment for covariates, including site-level con- 
text data from the ACT survey, will allow us to assess 
change in the client outcomes that are reported in the 
feedback report over time. For the primary analysis for 
this study, we will pool the RAI-HC data from all 
study sites, using RAI-HC data for six months prior to 
the intervention, during the intervention, and for six 
months after the intervention. We will assess changes 
in trends seen before, during, and after the interven- 
tion period for the client outcomes included in the 
feedback reports (pain, delirium, negative mood, falls, 
visits to hospital within previous 90 days) and other 
outcomes not included in the reports {e.g., pressure 
ulcers, incontinence, and social engagement). For the 
purpose of the time-series analysis, the quarterly data 
will be converted to monthly measures of aggregate 
client outcomes across all sites. Predictor variables will 
include the intervention dose, operationalized as the 
proportion of staff who were observed or who self- 
reported reading the reports. Data for control facilities 
will be requested and analyzed at the end of the post- 
surveillance period. 

Qualitative data from the focus groups and from the 
narrative observation notes will be coded into themes, 
including specific barriers and facilitators to feedback 
report uptake. At the site level, these findings will be 
used alongside the post-feedback report survey data, the 
client outcome data, and the findings from the ACT 
tool to glean additional detail around contextual differ- 
ences that may explain possible variations in responses 
to the feedback reports across sites. 

Timeline 

The DICE-LTC was completed in February 2010 [1]. 
The second phase of DICE, implementing a feedback 
intervention in HC/SL settings using the RAI-HC, began 
April 2011. 

Following a year-long intervention with quarterly 
report distribution to several HC offices/SL sites, DICE 
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Figure 1 This figure identifies the detailed project timeline for the DICE-HC study 



will enter its final year, focusing on dissemination and 
spread of the intervention throughout the province of 
Alberta. The full project timeline is provided in Figure 1. 

Dissemination and spread 

The final phase of the study is focused on spread of 
approaches to using the RAI data to inform decision 
making in HC/SL contexts and will mirror dissemina- 
tion activities used in the LTC phase [1]. In the final 
year of the DICE program, we will develop toolkits and 
training materials and will continue to work closely with 
the health region and managers within HC/SL settings 
that have expressed interest in continuing their engage- 
ment with a network of RAI users. After approaching 
zone leaders to obtain zone-level approval, we will con- 
tact administrators in HC/SL offices to request their 
voluntary participation. Our primary purpose is to offer 
the RAI coordinators in each of the settings tools, train- 
ing, and technical support to implement their own local 
program of feedback report generation and distribution. 
This implementation effort will be evaluated using a 
one-time survey in each participating setting to assess 
response to the feedback reports from all staff. We will 
also request historical RAI-HC data to assess any change 
in outcomes from the year prior to the implementation 
to six months after the training. This monitoring period 
is shorter than ideal and will be extended if we can 
secure additional funding. 

We believe this work to be relevant, timely, and 
occurring within a context where synergistic activities 
will support our dissemination plans. For example, Put- 
ting RAI to Work: Network of RAI Data Users and 
Researchers is a network of health authority representa- 
tives and organizations, funded through the Canadian 
Institutes for Health Research from 2008 to 2010, who 



have expressed interest in continuing their engagement 
in this network and in future work that focuses on 
using RAI data. They currently communicate their work 
with the RAI via an established website http://www.rair- 
esun.ca/. 

Deliverables 

The specific deliverables for this project will include (1) 
delineation of a process for identifying priorities across 
provider groups in continuing care settings in order to 
identify the core feedback report content, (2) a toolkit 
(manual and programming guides) for creating action- 
able feedback reports from the RAI data in HC/SL con- 
texts, (3) maintenance of the website to connect the 
network of decision makers and researchers interested 
in using RAI data for developing and delivering feedback 
reports, and (4) a group of decision makers and 
researchers who have the skills and support to develop 
and use tools in HC/SL sites [1]. 

As previously reported, we believe our work will be an 
important contribution to both policy and practice in 
continuing care contexts beyond Alberta. In addition to 
providing important guidance about use of feedback 
reports in care settings, our highly structured approach 
may provide some guidance to researchers in implemen- 
tation science in terms of organizing and planning 
audit-with-feedback interventions. 

Additional material 



Additional file 1: Survey instrument. This file contains an example of 
the post-feedback survey instrument. 

Additional file 2: Study team. This file contains the project team 
member descriptions at the time of implementation of DICE Phase II- 
Home care and Supportive Living. 
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Additional file 3: Feedback report. This file contains an example of the 
feedback report used in Home care/Supportive living sites. 
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